Cooling of data centers

ABSTRACT

In a cooling system for cooling racks in a data center, cooling fluid is circulated in the data center by a cooling device having a fan. In addition, this system includes a plenum having a plurality of returns and an outlet. The outlet of the plenum is in fluid communication with the fan and the plurality of returns are configured for removing the cooling fluid from the data center. Furthermore, the returns and are operable to vary a characteristic of the removal of the cooling fluid.

BACKGROUND

[0001] A data center may be defined as a location, e.g., room, thathouses numerous printed circuit (PC) board electronic systems arrangedin a number of racks. A standard rack may be defined as an ElectronicsIndustry Association (EIA) enclosure, 78 in. (2 meters) wide, 24 in.(0.61 meter) wide and 30 in. (0.76 meter) deep. Standard racks may beconfigured to house a number of PC boards, e.g., about forty (40)boards, with future configurations of racks being designed toaccommodate up to eighty (80) boards. The PC boards typically include anumber of components, e.g., processors, micro-controllers, high speedvideo cards, memories, semi-conductor devices, and the like, thatdissipate relatively significant amounts of heat during the operation ofthe respective components. For example, a typical PC board comprisingmultiple microprocessors may dissipate approximately 250 W of power.Thus, a rack containing forty (40) PC boards of this type may dissipateapproximately 10 KW of power.

[0002] The power required to remove the heat dissipated by thecomponents in the racks is generally equal to about 10 percent of thepower needed to operate the components. However, the power required toremove the heat dissipated by a plurality of racks in a data center isgenerally equal to about 50 percent of the power needed to operate thecomponents in the racks. The disparity in the amount of power requiredto dissipate the various heat loads between racks and data centers stemsfrom, for example, the additional thermodynamic work needed in the datacenter to cool the air. In one respect, racks are typically cooled withfans that operate to move cooling fluid, e.g., air, across the heatdissipating components; whereas, data centers often implement reversepower cycles to cool heated return air. The additional work required toachieve the temperature reduction, in addition to the work associatedwith moving the cooling fluid in the data center and the condenser,often add up to the 50 percent power requirement. As such, the coolingof data centers presents problems in addition to those faced with thecooling of racks.

[0003] Conventional data centers are typically cooled by operation ofone or more air conditioning units. The compressors of the airconditioning units typically require a minimum of about thirty (30)percent of the required cooling capacity to sufficiently cool the datacenters. The other components, e.g., condensers, air movers (fans),etc., typically require an additional twenty (20) percent of therequired cooling capacity. As an example, a high density data centerwith 100 racks, each rack having a maximum power dissipation of 10 KW,generally requires 1 MW of cooling capacity. Air conditioning units witha capacity of 1 MW of heat removal generally requires a minimum of 300KW input compressor power in addition to the power needed to drive theair moving devices, e.g., fans, blowers, etc. Conventional data centerair conditioning units do not vary their cooling fluid output based onthe distributed needs of the data center. Instead, these airconditioning units generally operate at or near a maximum compressorpower even when the heat load is reduced inside the data center.

[0004] The substantially continuous operation of the air conditioningunits is generally designed to operate according to a worst-casescenario. That is, cooling fluid is supplied to the components at around100 percent of the estimated cooling requirement. In this respect,conventional cooling systems often attempt to cool components that maynot be operating at a level which may cause their temperatures to exceeda predetermined temperature range. In addition, conventional returnsystems remove air from the data centers in an indiscriminate manner.That is, conventional return systems may remove relatively cool air fromdata centers and/or may not efficiently remove relatively warm air fromdata centers. Consequently, conventional cooling systems often incurgreater amounts of operating expenses than may be necessary tosufficiently cool the heat generating components contained in the racksof data centers.

SUMMARY

[0005] In accordance with an embodiment, the invention pertains to acooling system for cooling racks in a data center. In this system,cooling fluid is circulated in the data center by a cooling devicehaving a fan. In addition, this system includes a plenum having aplurality of returns and an outlet. The outlet of the plenum is in fluidcommunication with the fan and the plurality of returns are configuredto remove the cooling fluid from the data center. Furthermore, thereturns and are operable to vary a characteristic of the removal of thecooling fluid.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the invention are illustrated by way of exampleand not limitation in the accompanying figures in which like numeralreferences refer to like elements, and wherein:

[0007]FIG. 1 shows a simplified schematic illustration of a data centercontaining a cooling system in accordance with an embodiment of theinvention;

[0008]FIGS. 2A and 2B illustrate block diagrams of respective controlschemes for cooling systems according to various embodiments of theinvention; and

[0009]FIGS. 3A and 3B show flow diagrams of a first and second manner inwhich embodiments of the invention may be practiced.

DETAILED DESCRIPTION

[0010] For simplicity and illustrative purposes, the principles of theinvention are described by referring mainly to an embodiment thereof. Inthe following description, numerous specific details are set forth inorder to provide a thorough understanding of the invention. It will beapparent however, to one of ordinary skill in the art, that theinvention may be practiced without limitation to these specific details.In other instances, well known methods and structures have not beendescribed in detail so as not to unnecessarily obscure the invention.

[0011] According to an embodiment of the invention, a cooling system isconfigured to adjust cooling fluid (e.g., air, other gasses, liquid,etc.) flow to and/or from various racks located throughout a datacenter, e.g., a location that houses numerous printed circuit (PC) boardelectronic systems arranged in a number of racks, based upon thedetected or anticipated temperatures at various locations throughout thedata center. In one respect, by substantially increasing the coolingfluid flow to and/or from those racks dissipating greater amounts ofheat and by substantially decreasing the cooling fluid flow to and/orfrom those racks dissipating lesser amounts of heat, the amount ofenergy required to operate the cooling system may be relatively reduced.Specifically, by modifying the return flow rate of cooling fluid,relatively hot fluid may be removed from the data center whilerelatively cooler fluid may be allowed to remain. Thus, instead ofoperating the devices, e.g., compressors, fans, etc., of the coolingsystem at substantially 100 percent of the anticipated heat dissipationfrom the racks, those devices may be operated according to the actualcooling needs. In addition, the racks may be positioned throughout thedata center according to their anticipated heat loads to thereby enablecomputer room air conditioning (CRAC) units located at various positionsthroughout the data center to operate in a more efficient manner. Inanother respect, the positioning of the racks may be determined throughimplementation of modeling and metrology of the cooling fluid flowthroughout the data center. In addition, the numerical modeling may beimplemented to determine the volume flow rate and velocity of thecooling fluid flow through each of the vents. Furthermore, the numericalmodeling may be implemented to determine the volume flow rate andvelocity of the return cooling fluid flow through each of the returnvents.

[0012]FIG. 1 shows a simplified schematic illustration of a data center10 containing a cooling system 12. The data center 10, according to thisembodiment of the invention, includes a raised floor 14. A plurality ofwires and communication lines (not shown) may be located in a space 16beneath the raised floor 14. In addition, the space 16 may function as aplenum to deliver cooling fluid (e.g., air) from the cooling system 12to a plurality of racks 18 a-18 d. The data center 10 may furtherinclude a lowered ceiling 20. In a manner similar to the raised floor14, the lowered ceiling 20 may include a space 22 in which a pluralityof wires and communication lines (not shown) may be located. Inaddition, the space 22 may function as a plenum to return air to thecooling system 12. Although the data center 10 is illustrated in FIG. 1as containing four racks 18 a-18 d and a cooling system 12, it should beunderstood that the data center may include any number of racks, e.g.,100 racks, and cooling systems, e.g., four or more. The illustration offour racks is for illustrative and simplicity of description purposesonly and is not intended to limit the invention in any respect.

[0013] The racks 18 a-18 d generally house a plurality of components(not shown), e.g., processors, micro-controllers, high speed videocards, memories, semi-conductor devices, and the like. The componentsmay be elements of a plurality of subsystems (not shown), e.g.,computers, servers, etc. The subsystems and the components may beimplemented to perform various electronic, e.g., computing, switching,routing, displaying, and the like, functions. In the performance ofthese electronic functions, the components, and therefore thesubsystems, generally dissipate relatively large amounts of heat.Because racks 18 a-18 d have been generally known to include upwards offorty (40) or more subsystems, they may transfer substantially largeamounts of heat to the cooling fluid to maintain the subsystems and thecomponents generally within a predetermined operating temperature range.

[0014] As this air is heated in the vicinity of the racks 18 a-18 d, itmay expand and create a localized area of relatively high pressure. Thismay inhibit movement of relatively cool incoming air. To compensate forthis phenomenon, conventional cooling systems may cool surrounding areasbelow the predetermined operating range. However, this may beinefficient because relatively more energy may be utilized to cool thesesurrounding areas below the predetermined operating range than wouldotherwise be required in a system with sufficient air movement.Furthermore, if the relatively hot air is drawn into the racks 18 a-18d, it may not have sufficient cooling potential to maintain thesubsystems and the components generally within the predeterminedoperating temperature range. Therefore, according to one aspect of anembodiment of the invention, by substantially controlling the amount ofheated cooling fluid (e.g., heated air, return air, etc.) removed fromthe general vicinity of the components and the subsystems located in theracks 18 a-18 d based upon their respective heat loads, the powerconsumed by the cooling system 12 to maintain the components atpredetermined operating temperatures may also be controlled.

[0015] The cooling system 12 generally includes a fan 24 for supplyingcooling fluid (e.g., air) into the space 16 (e.g., plenum) and/ordrawing air from the space 22 (e.g., plenum). Air, heated in the datacenter 10 is supplied to the fan 24 from the space 22 as indicated by anarrow 26. In operation, the heated air (e.g., return air) enters intothe cooling system 12 as indicated by the arrow 26 and is cooled byoperation of a cooling coil 28, a compressor 30, and a condenser 32, inany reasonably suitable manner generally known to those of ordinaryskill in the art. In terms of cooling system efficiency, it is generallydesirable that the return air is composed of the relatively warmestportion of air in the data center 10.

[0016] Although reference is made throughout the present disclosure ofthe use of a fan 24 to draw heated air from the space 22, it should beunderstood that any other reasonably suitable manner of air removal maybe implemented without departing from the scope of the invention. By wayof example, a separate fan (not shown) may be provided to draw air fromthe space 22. Alternatively, the intakes of the space 22 may be providedwith fans (not shown) such that each intake may draw a substantiallyindependent amount of air from the data center 10.

[0017] In addition, based upon the cooling fluid needed by the heatloads in the racks 18 a-18 d, the cooling system 12 may be operated atvarious levels. For example, the capacity (e.g., the amount of workexerted on the refrigerant) of the compressor 30 and the speed of thefan 24 may both be modified to thereby control the temperature and theamount of cooling fluid flow delivered to the racks 18 a-18 d. In thisrespect, the compressor 30 is a variable capacity compressor and the fan24 is a variable speed fan. The compressor 30 may thus be controlled toeither increase or decrease the mass flow rate of a refrigeranttherethrough. Because the specific type of compressor 30 and fan 24 tobe employed with the embodiments of the invention may vary according toindividual needs, the invention is not limited to any specific type ofcompressor or fan. Instead, any reasonably suitable type of compressor30 and fan 24 that are capable of accomplishing certain aspects of theinvention may be employed with the embodiments of the invention. Thechoice of compressor 30 and fan 24 may depend upon a plurality offactors, e.g., cooling requirements, costs, operating expenses, etc.

[0018] The cooling fluid generally flows from the fan 24 and into thespace 16 (e.g., plenum) as indicated by the arrow 34. The cooling fluidflows out of the raised floor 14 through a plurality of dynamicallycontrollable vents 36 a-36 c that generally operate to control thevelocity and the volume flow rate of the cooling fluid therethrough. Amore detailed description of the above-described embodiment may be foundin co-pending U.S. application Ser. No. 09/970,707, filed Oct. 5, 2001,which is assigned to the assignee of the present invention and isincorporated by reference herein in its entirety.

[0019] The cooling fluid may absorb dissipated heat as it flows throughthe racks 18 a-18 d. This heated cooling fluid may flow into the loweredceiling through a plurality of dynamically controllable returns 38 a-38c that generally operate to control the velocity, direction, and thevolume flow rate of the heated cooling fluid therethrough. In onerespect, the velocity and the volume flow rate of the cooling fluid maybe regulated by varying the shape and/or opening size of the vents 36a-36 c and/or the returns 38 a-38 c. In another respect, the directionfrom which the cooling fluid is received may also be varied. Forexample, louvers within the returns 38 a-38 c (not specifically shown inFIG. 1) may be modified to vary the intake of the heated cooling fluidtherethrough. In yet another respect, fans within the returns 38 a-38 c(not specifically shown in FIG. 1) may be configured to vary the volumeflow rate and/or velocity of the heated cooling fluid therethrough.Thus, according to this embodiment of the invention, the racks 18 a-18 dmay receive substantially individualized and localized amounts ofcooling fluid according to their heat loads. In addition, the removal ofthe heated cooling fluid in the vicinity of the racks 18 a-18 d may besubstantially individualized according to the localized heat load.

[0020] The arrows 40 indicate the general direction of travel of thecooling fluid and the dashed arrows 42 indicate the general direction oftravel of fluid heated by the heat dissipating components located withinthe racks 18 a-18 d. As may be seen in FIG. 1, the areas between theracks 18 a-18 d may comprise either cool aisles 44 or hot aisles 46, ora combination thereof. The cool aisles 44 are those aisles that includethe vents 36 a-36 c and thus receive cooling fluid for delivery to theracks 18 a-18 d. The hot aisles 46 are those aisles that receive airheated by the heat dissipating components in the racks 18 a-18 d. Thereturns 38 a-38 c may be positioned to remove air from the hot aisles46. In this regard, the returns 38 a-38 c may remove relatively moreheated fluid from the hot aisles 46. Thus, reducing the lifetime ofrelatively hot air particles within the data center 10 and thelikelihood of these hot air particles being drawn back into the racks 18a-18 d.

[0021] In addition, various sections of each of the racks 18 a-18 d mayalso receive substantially individualized amounts of cooling fluid. Byway of example, if the bottom halves of the racks 18 a and 18 b areoperating at maximum power, thereby dissipating a maximum level of heatload, and the upper halves are operating at little or no power, the vent36 c, the return 38 c, and/or the return 38 b may be configured toenable cooling fluid flow therethrough to have a relatively high volumeflow rate with a relatively low velocity. In this manner, the coolingfluid may operate to generally supply greater cooling to the lowerhalves of the racks 18 a and 18 b, whereas the upper halves receiverelatively lesser amounts of cooling fluid. In addition, if the upperhalves of the racks 18 c and 18 d are operating at approximately 50percent of their maximum power, and the lower halves are operating atlittle or no power, the vent 34 b and/or the return 38 a may beconfigured to enable cooling fluid flow therethrough to have arelatively low volume flow rate with a relatively high velocity. In thismanner, the cooling fluid flow may have sufficient momentum toadequately reach and cool the upper halves of the racks 18 c and 18 d.

[0022] Moreover, as the cooling requirements vary according to the heatloads in the racks 18 a-18 d, and the subsequent variations in thevolume flow rate of the cooling fluid, the cooling system 12 may alsovary the amount of cooling fluid supplied to the racks. As an example,if the heat load in the racks 18 a-18 d generally increases, the coolingsystem 12 may operate to increase the supply of cooling fluid and/or thereturn of heated fluid. Alternatively, if the heat load in the racks 18a-18 d generally decreases, the cooling system 12 may operate todecrease the supply of cooling fluid and/or the return of heated fluid.The vents 36 a-36 c thus generally provide localized control of thecooling fluid flow to the racks 18 a-18 d and the returns 38 a-38 cgenerally provide localized control of the heated fluid flow back to thecooling system 12. In this regard, the cooling system 12 generallyprovides global control of the cooling fluid flow and/or the heatedfluid flow. In one respect, therefore, the amount of energy consumed bythe cooling system 12 in maintaining the racks 18 a-18 d at apredetermined temperature range may be substantially reduced incomparison with conventional data center cooling systems.

[0023] According to an embodiment of the invention, the flow of heatedfluid through the returns 38 a-38 c may be modified in response to thetemperature of the fluid at and/or near each of the returns 38 a-38 c.In this respect, each of the returns 38 a-38 c may include a respectivetemperature sensor 48 a-48 c. For example, in response to thetemperature sensor 48 a detecting a high temperature, relative to apredetermined temperature and/or the other temperature sensors 48 b-48c, the return 38 a may be modified to increase the flow of heated fluidtherethrough.

[0024] In addition, the capacity of the compressor 30 may vary accordingto changes in the temperature of the returned heated fluid. As such, thetemperature sensors 48 a-48 c may relay temperature measurements to thecooling system 12. The temperature sensors 48 a-48 c may comprise anyreasonably suitable temperature sensor known to those skilled in theart. Therefore, the compressor 30 may be operated to generally maintainthe temperature of the heated fluid within each of the returns 38 a-38 cat a substantially constant level. In addition, the capacity of thecompressor 30 may also vary according to detected and/or anticipatedchanges in heat loads generated in the racks 18 a-18 d, the flow ratesof the vents 36 a-36 c and/or the returns 38 a-38 c, and/or varioussensed pressures within the data center 10. As an example, thecompressor 30 capacity may be increased as the heat loads generated inthe racks 18 a-18 d increase. In this regard, the power required tooperate the compressor 30 may be substantially optimized, therebyreducing the total power required to operate the cooling system 12.

[0025] According to an embodiment of the invention, the flow of heatedfluid through the returns 38 a-38 c may be modified in response to theflow of the cooling fluid supply through the vents 36 a-36 c. Forexample, in response to the vent 36 a being modified to increase theflow of the cooling fluid therethrough, the return 38 a may, in asimilar manner, be modified to increase the flow of heated fluid. By wayof example, the flow rates of the cooling fluid across correspondingvents 36 a-36 c and returns 38 a-38 c may be measured (e.g., eitherdirectly or by measuring pressure drop and using a suitable correlation)and synchronized. The flow rates of the vents 36 a-36 c may bedetermined by measuring temperature in the room, for instance, and theflow rates of the returns 38 a-38 c can be set to equal the flow ratesof the vents 36 a-36 c. If the flow rates are relatively matched, orsome correlation between them utilized, the recirculation of hot air inthe room may be reduced as well as mixing of the hot air and the coolercooling fluid, to thereby improve system efficiency. In this respect,the returns 38 a-38 c may control the flow of return air based on theoperation of the vents 36 a-36 c. Thus, only that amount of energyrequired to substantially cool the components contained in the racks 18a-18 d may be expended, which may correlate to a substantial energysavings over known cooling systems.

[0026] The capacity of the compressor 30 may vary according to changesin the temperature of the return air located in the space 22. As such, aplenum temperature sensor 50 may be located within the space 22 to relaytemperature measurements to the cooling system 12. The plenumtemperature sensor 50 may comprise any reasonably suitable temperaturesensor known to those skilled in the art. Therefore, the compressor 30may be operated to generally maintain the temperature of the return airwithin the space 22 at a substantially constant level. Similarly, it iswithin the scope of the invention that the capacity of the compressor 30may vary according to changes in the temperature of the air located inthe space 16. As such, a plenum temperature sensor 52 may be locatedwithin the space 16 to relay temperature measurements to the coolingsystem 12. In addition, the capacity of the compressor 30 may also varyaccording to detected and/or anticipated changes in heat loads generatedin the racks 18 a-18 d. As an example, the compressor 30 capacity may beincreased as the heat loads generated in the racks 18 a-18 d increase.In this regard, the power required to operate the compressor 30 may besubstantially optimized, thereby reducing the total power required tooperate the cooling system 12.

[0027] As discussed above, the fan 24 is a variable speed fan. In thisregard, it is within the scope of the invention that the speed of thefan 24 be modifiable based on a variety of factors. For example, in anembodiment of the invention, a pressure sensor 56 may be configured tomeasure the pressure of the returning cooling fluid in the space 22 andrelay these measurements to the cooling system 12. In anotherembodiment, a pressure sensor 58 may be configured to measure thepressure of the cooling fluid in the space 16 and relay thesemeasurements to the cooling system 12. Based on the pressuremeasurements from the pressure sensor 56 and/or 58 the speed of the fan24 may be varied. In this manner, the power required to operate the fan24 may be substantially optimized, thereby reducing the total powerrequired to operate the cooling system 12.

[0028] In addition, the discussion above describes the inclusion oftemperature and pressure sensors. However, it is within the scope ofvarious embodiments of the invention that any reasonable type of sensorbe included. In general, these sensors may be operable to sense ormeasure environmental conditions (e.g., temperature, pressure, humidity,wind speed, etc.) and relay data related to the sensed (or measured)environmental condition to the cooling system 12. Specific environmentalconditions which these types of sensors may be utilized to detectinclude, but are not limited to: temperature, pressure, humidity, andfluid flow rate. These sensors may be positioned at various locationswithin the data center 10. For example, sensors may be placed in theracks, vents, returns, plenums, or the like.

[0029] Referring to FIG. 2A, there is illustrated a block diagram 200 ofa control scheme for a cooling system 202 according to an embodiment ofthe invention. The following description of the block diagram 200 is onemanner in which the cooling system 202 may be operated. In this respect,it is to be understood that the following description of the blockdiagram 200 is but one manner of a variety of different manners in whichsuch a cooling system 202 may be operated. According to this embodimentof the invention, the cooling system 202 includes a return unit 204, avent unit 206, and a heat exchange unit 208.

[0030] The return unit includes a return controller 210 generallyconfigured to control the operation of returns 38 a-38 c. In thisregard, the return controller 210 may comprise a microprocessor, amicro-controller, an application specific integrated circuit (ASIC), andthe like. In an embodiment of the invention, the manner in which thereturn controller 210 operates the returns 38 a-38 c, i.e., the flow ofreturn air therethrough, may be predicated upon the detected oranticipated temperatures of the racks 18 a-18 d or portions thereof. Forexample, with regard to detected temperatures, a plurality oftemperature sensors 48 a-48 c, e.g., thermocouples, may be positioned atvarious positions around the subsystems and/or the racks 18 a-18 d. Eachof the temperature sensors 48 a-48 c may correspond to a respective oneof the returns 38 a-38 c. By way of example, one temperature sensor 48 amay affect the return flow of cooling fluid flow through one return 38a. Alternatively, with regard to anticipated temperatures, anticipatedcooling requirements for each of the racks 18 a-18 d and/or varioussections of the racks may be predicated upon an impending load on theracks 18 a-18 d and/or sections of the racks. For example, the returncontroller 210 may be connected to another controller, e.g., a centralcontroller for the subsystems, which anticipates the heat load thecomponents and/or the subsystems will dissipate. This information may berelayed to the return controller 210 which may then manipulate thereturns 38 a-38 c according to the anticipated load.

[0031] In addition to and/or in another embodiment of the invention, themanner in which the return controller 210 operates the returns 38 a-38 cmay be predicated upon the detected (e.g., sensed, measured, etc.) orcalculated flow rate of the vents 36 a-36 c or portions thereof. Forexample, with regard to detected flow rates, a plurality of flow sensors(not shown), may be positioned in or near the vents 36 a-36 c. Each ofthe flow sensors may correspond to a respective one of the returns 38a-38 c and in a manner similar to above, may affect the return flow ofcooling fluid flow through one return 38 a-38 c. Alternatively, the flowrate may be calculated by correlating another sensed environmentalcondition, such as pressure change across each vent 36 a-36 c, with theflow rate. This information may be relayed to the return controller 210which may then manipulate the returns 38 a-38 c according to thecalculated flow.

[0032] Although FIG. 2A illustrates three temperature sensors 48 a-48 cconnected to the return controller 210, it should be understood that thenumber of temperature sensors is not critical to the operation of thevarious embodiments of the invention. Instead, the cooling system 202may include any reasonably suitable number of temperature sensors tothus measure the temperatures of any reasonably suitable number of racks18 a-18 d or portions thereof. The number of temperature sensors and thetemperature measurements of the number of racks may be upgradable, e.g.,scalable, to include any additional components and/or racks that may beincluded in the data center. In addition, the temperature sensors neednot be stationary. In this regard, according to another embodiment ofthe invention, a mobile device (not shown) is implemented to gather ormeasure at least one local environmental condition (e.g., temperature,pressure, air flow, humidity, etc.) in the data center 10. Moreparticularly, the mobile device is configured to travel around the racksto determine the one or more environmental conditions at variouslocations throughout the data center. In addition, the device may beconfigured to detect the one or more environmental conditions at variousheights throughout the data center. The information gathered by themobile device may be transmitted to the cooling system 202. As describedhereinbelow, a controller within the cooling system 202 may vary thedelivery and temperature of cooling fluid according to the one or moredetected environmental conditions. In this respect, the energy necessaryto cool the racks and the components contained therein, maysubstantially be optimized.

[0033] A more detailed description of the above-described embodiment maybe found in co-pending U.S. application Ser. No. 10/157,892, filed May31, 2002, which is assigned to the assignee of the present invention andis incorporated by reference herein in its entirety.

[0034] If there is an actual detected change or an anticipated change inthe temperature of the respective racks 18 a-18 d and/or portionsthereof, the return controller 210 generally operates to manipulate thecorresponding return 38 a-38 c to compensate, i.e., changes the volumeflow rate, velocity, and other similar characteristics of the coolingfluid, for the change in temperature. In this respect, heated coolingfluid may be removed from the vicinity of each of the racks 18 a-18 dand/or portions thereof substantially only as necessary to maintain thetemperature of the portions of the racks within a predeterminedtemperature range. As will be seen from the discussion hereinbelow, bycontrolling the cooling fluid flow in this manner, the compressors 30and fans 24 may be operated at substantially optimized levels, therebydecreasing the amount of energy and thus the operating costs required tooperate these devices.

[0035] Return interface electronics 224 may be provided to act as aninterface between the return controller 210 and the components, e.g.,control the opening in the returns 38 a-38 c and the return flow throughthe returns 38 a-38 c.

[0036] The return controller 210 may also be interfaced with a returnmemory 226 configured to provide storage of a computer software thatprovides the functionality of the cooling system and may be executed bythe return controller 210. The memory 226 may also be configured toprovide a storage for containing data/information pertaining to themanner in which each of the returns 38 a-38 c may be manipulated inresponse to the detected and/or anticipated temperatures of the portionsof the racks 18 a-18 d. In keeping with the example cited hereinabove,the return controller 210 may operate the return 38 a to increase thevolume flow rate and decrease the velocity of the cooling fluid flowingtherethrough in response to a detected increase in the heat load of alower portion of a corresponding rack. The memory 226 may be implementedas a combination of volatile and non-volatile memory, such as dynamicrandom access memory (DRAM), EEPROM, flash memory, and the like.

[0037] The vent unit 206 may be configured to operate in a mannersimilar to the return unit 204. In this regard, the vent unit 206 mayinclude a vent controller 228, vents 36 a-36 c, temperature sensors236-240, vent interface electronics 242 and a memory 244. With respectto the temperature sensors 236-240, in an embodiment of the invention,these sensors may be the temperature sensors 48 a-48 c configured torelay temperature measurements to both the return controller 210 and thevent controller 228. In various other embodiments of the invention, thetemperature sensors 236-240 may be distinct from the temperature sensors48 a-48 c. In these various other embodiments, the temperature sensors236-240 may be placed in or around the racks 18 a-18 d and/or the vents36 a-36 c.

[0038] The vent controller 228 and the return controller 210 may beconfigured to relay data/information pertaining to temperaturemeasurements and/or the flow of cooling fluid to each other and/or tothe heat exchange unit 208. More specifically, the heat exchange unit208 may include a cooling system controller 246 configured tocommunicate with the return controller 210 and/or the vent controller228. The cooling system controller 246 is generally configured tocontrol the operation of the cooling system 12, e.g., the compressor 30and the fan 24. In this regard, the controller 228 may comprise amicroprocessor, a micro-controller, ASIC, and the like.

[0039] Interface electronics 252 may be provided to act as an interfacebetween the cooling system controller 246 and the components foroperating the compressor 30 and the fan 24, e.g., the supply of voltageto vary the respective speeds of the compressor and the fan, directcontrol of the compressor and the fan, etc.

[0040] The cooling system controller 246 may also be interfaced with amemory 254 configured to provide storage of a computer software thatprovides the functionality of the cooling system 12, e.g., compressor 30and fan 24, and may be executed by the cooling system controller 246.The memory 254 may also be configured to provide a storage forcontaining data/information pertaining to the manner in which thecompressor 30 and the fan 24 may be manipulated in response tovariations in the return fluid flow through the returns 38 a-38 c and/orfluid flow through the vents 36 a-36 c. In keeping with the examplecited hereinabove, the cooling system controller 246 may operate thecompressor 30 and the fan 24 to increase/decrease the volume flow rateof the cooling fluid flow in response to various degrees of detectedincreases/decreases in the volume flow rate through the returns 38 a-38c and/or the vents 36 a-36 c. More particularly, a look up table (notshown) may be stored in the memory 254. By way of example, the look uptable may include information pertaining to the level of compressor 30speed and fan 24 output increase necessary for a detected increase inthe volume flow rate. In this respect, the compressor 30 speed and thefan 24 output may be varied substantially incrementally in response todetected changes in the volume flow rate. The memory 254 may beimplemented as a combination of volatile and non-volatile memory, suchas dynamic random access memory (DRAM), EEPROM, flash memory, and thelike.

[0041] Although FIG. 2A illustrates a single return controller 210configured to operate the returns 38 a-38 c, it should be understoodthat a plurality of return controllers may be implemented to perform thefunctions of the return controller 210 without deviating from the scopeand spirit of the invention.

[0042] In FIG. 2B, there is illustrated a block diagram 260 of anothercontrol scheme for a cooling system 202 according to the invention. Theelements illustrated in the block diagram 260 operate in substantiallythe same manner as those elements illustrated in the block diagram 200.However, one difference lies in the substantially independentoperability of the heat exchange unit 208 from the return unit 204and/or the vent unit 206. That is, operation of the cooling systemcontroller 246 may not be directly related to the operation of thereturn controller 210 and/or the vent controller 228. Because of theapparent similarities between the block diagrams 200 and 260, only thoseelements that differ between the block diagrams will be describedhereinbelow.

[0043] Pressure sensors 56-58 may be configured to measure the pressurewithin the space 16 and/or space 22 (e.g., plenums) as describedhereinabove. The pressure measurements and/or any discernable changes inthe pressure measurements obtained by the pressure sensor(s) 56-58 maybe relayed to the cooling system controller 246. In addition, at leastone plenum temperature sensor 50-52 may be configured to measure thetemperature of the fluid within the space 16 and/or space 22. Thetemperature measurements and/or any discernable changes in thetemperature obtained by the plenum temperature sensor may also berelayed to the cooling system controller 246.

[0044] The cooling system controller 246 may manipulate the capacity ofthe compressor 30 based upon the measured temperature of the fluid. Thatis, the temperature of the fluid within the space 16 and/or space 22 maybe maintained at a substantially constant level by manipulation of thecompressor. Further, the output of the fan 24 may be manipulated basedupon the measured pressure of the fluid in the space 16 to vary theamount of cooling fluid supplied to space 16, to thereby substantiallymaintain the pressure of the cooling fluid within the space 16 at asubstantially uniform level. Similarly, in addition to or as analternative to manipulating the fan 24 output in response to pressuremeasurements within the space 16, these manipulations may be based uponpressure measurements within the space 22. Thus, the cooling systemcontroller 246 is operable to increase the speed of the compressor 30and the fan 24 output, e.g., expend a greater amount of energy,substantially as the heat loads in the racks 18 a-18 d requires such anincrease. Consequently, the compressor 30 and the fan 24 are notoperated at a substantially constant energy level and the amount ofenergy necessary is substantially lower than that of conventionalcooling systems that typically operate at maximum energy levels.

[0045] The memory 254 may also be configured to store data/informationpertaining to the control of the compressor 30 speed and the output ofthe fan 24 corresponding to the measured pressure with the space 16and/or space 22. For example, the cooling system controller 246 mayincrease the compressor 30 speed and fan 24 output by a relatively largeamount in response to a relatively large decrease in the measuredpressure. In this respect, the pressure within the space 16 and/or space22 may be maintained at a substantially uniform level even when thepressures change by a relatively sharp amount.

[0046]FIG. 3A shows a flow diagram 300 of a first manner in which anembodiment of the invention may be practiced. The following descriptionof the flow diagram 300 is made with reference to the block diagram 200illustrated in FIG. 2A, and thus makes reference to the elements citedtherein. It is to be understood that the steps illustrated in the flowdiagram 300 may be contained as a utility, program, subprogram, in anydesired computer accessible medium. In addition, the flow diagram 300may be embodied by a computer program, which can exist in a variety offorms both active and inactive. For example, they can exist as softwareprogram(s) comprised of program instructions in source code, objectcode, executable code or other formats. Any of the above can be embodiedon a computer readable medium, which include storage devices andsignals, in compressed or uncompressed form.

[0047] Examples of computer readable storage devices includeconventional computer system RAM (random access memory), ROM (read onlymemory), EPROM (erasable, programmable ROM), EEPROM (electricallyerasable, programmable ROM), and magnetic or optical disks or tapes.Examples of computer readable signals, whether modulated using a carrieror not, are signals that a computer system hosting or running thecomputer program can be configured to access, including signalsdownloaded through the Internet or other networks. Concrete examples ofthe foregoing include distribution of the programs on a CD ROM or viaInternet download. In a sense, the Internet itself, as an abstractentity, is a computer readable medium. The same is true of computernetworks in general. It is therefore to be understood that thosefunctions enumerated below may be performed by any electronic devicecapable of executing the above-described functions.

[0048] In the flow diagram 300, the cooling system 202 is activated andthe vents 36 a-36 c and the returns 38 a-38 c are opened at step 302.The temperature of a component (Tc) generally corresponds to the heatload of the heat dissipating components and therefore the subsystemscontained in the racks 18 a-18 d. Therefore, the Tc's may be based uponthe temperatures of specific heat dissipating components and subsystems.In addition, the Tc's may be based upon the temperatures in the generalvicinity of the racks and/or sections of the racks. Thus, those skilledin the art will understand that certain embodiments of the invention maybe employed with the temperature sensors (e.g., 48 a-48 c, 50, 52,236-240, etc.) located at various positions throughout the data center.Furthermore, it is within the scope of the invention that thetemperature be sensed via mobile temperature sensors. Moreover, use ofthe term “rack” herein generally refers additionally to sections of theracks and thus may not necessarily refer to an entire rack. Thus, theuse of the term “rack” throughout the disclosure is not meant to limitcertain aspects to entire racks, but instead, is relied upon to simplifythe description of certain embodiments of the invention.

[0049] At step 304, the temperatures of the components (Tc's) areindividually sensed by the temperature sensors. Alternatively, the Tc'smay be anticipated in the manner described hereinabove with respect toFIG. 2A. At step 306, it is determined whether each of the measuredtemperatures are individually within a predetermined range of operatingtemperatures, e.g., between a maximum set point temperature (Tmax,set)and a minimum set point temperature (Tmin,set). The predetermined rangeof operating temperatures may be set according to a plurality offactors. These factors may include, for example, the operatingtemperatures set forth by the manufacturers of the subsystems andcomponents located in the racks, through testing to determine theoptimal operating temperatures, etc. In addition, the predeterminedrange of operating temperatures may vary from one subsystem to anotheron the basis that various subsystems generally may operate effectivelyat various temperatures.

[0050] The measured and/or anticipated temperatures for those racksdetermined to have heat loads that fall within the predetermined rangeof operating temperatures, are sensed again at step 304. For those racksdetermined to have heat loads that do not fall within the predeterminedtemperature range, i.e., fall outside of Tmin,set and Tmax,set, it isdetermined whether the sensed temperature equals or falls below theTmin,set at step 308. In general, the range of temperatures Tmin,set andTmax,set pertains to threshold temperatures to determine whether toincrease or decrease the flow of cooling fluid delivered to the racks.The predetermined temperature range may be based upon a plurality offactors, for example, a threshold operating range of temperatures thatmay be determined through testing to substantially optimize theperformance of the subsystems contained in the racks. Moreover, thepredetermined temperature range may vary for each rack because variouscomponents generally may operate effectively at various temperatures andthus various threshold temperatures may be optimal.

[0051] If the Tc's of some of the racks are below or equal to theTmin,set, the return controller 210 and/or the vent controller 228 mayoperate to decrease the volume flow rate and/or the velocity of coolingfluid circulating about those racks at step 310. The determination ofwhether to decrease either or both the volume flow rate and the velocityof the cooling fluid may be based upon the detected temperature of theracks. For example, if the subsystems on a bottom half of a rack areoperating at 50 percent of maximum capacity, and the subsystems on anupper half of the rack are operating at or near zero capacity, thevelocity of the cooling fluid may be reduced whereas the volume flowrate may remain substantially constant. This may occur, for example,because the cooling fluid need not travel a relatively long distance butmay still need to supply the bottom half with a sufficient amount ofcooling fluid.

[0052] If the Tc's of some of the racks exceed the Tmin,set (i.e., alsoexceed the Tmax,set), the return controller 210 and/or the ventcontroller 228 may operate to increase the volume flow rate and/or thevelocity of cooling fluid circulating about those racks at step 312. Thedetermination of whether to increase either or both the volume flow rateand the velocity of the cooling fluid may be based upon the detectedtemperature of the racks. For example, if the subsystems on the top halfof a rack are operating at 100 percent capacity, and the subsystems on abottom half of the rack are operating at or near zero capacity, thevelocity and the volume flow rate of the cooling fluid may both beincreased. This may occur, for example, because the cooling fluid musttravel a relatively long distance and supply the top half with asufficient amount of cooling fluid.

[0053] According to an embodiment of the invention, the decrease involume flow rate and/or velocity of the cooling fluid flow at step 310and the increase in volume and/or velocity of the cooling fluid at step312 may be accomplished by incrementally varying the cooling fluid flowthrough the returns 38 a-38 c and/or the vents 36 a-36 c. An examplewill be made for the instance where a return allows a certain amount ofcooling fluid to flow therethrough, and the return is manipulated toincrease the volume flow rate of the cooling fluid, and where theincrease in fluid flow is insufficient to cause the Tc for that rack tofall within the predetermined range. In this instance, during asubsequent run through steps 204-210, the return may be controlled tofurther increase the volume flow rate of the cooling fluid therethroughby an incremental amount. By repeating this process a number of times,the temperature of the rack may be substantially brought within thepredetermined range. Similarly, manipulation of a vent in conjunctionwith the manipulation of the return may allow for greater control of thevolume flow rate of fluid in and/or around a rack. In this manner,greater temperature control may be facilitated and greater energysavings may be realized.

[0054] At step 314, the cooling system controller 246 may determinewhether to decrease the cooling fluid intake, e.g., decrease the speedof the compressor 30 and/or the fan 24. The determination of whether todecrease the cooling fluid intake may be made in response to themanipulations made to the returns 38 a-38 c by the return controller210. For instance, if the total amount of decreases in the volume flowrates of the cooling fluid exceeds the total amount of increases in thevolume flow rates flow of the cooling fluid, the cooling systemcontroller 246 may operate to decrease the cooling fluid intake at step316. Alternatively, if the total amount of increases in the volume flowrates of the cooling fluid exceeds the total amount of decreases, thecooling system controller 246 may operate to increase the cooling systemintake at step 318.

[0055] Following steps 316 or 318, or if the increases in the volumeflow rates of the cooling fluid through the returns equals thedecreases, for example, the Tc's are sensed again at step 304. Inaddition, the steps following step 304 may be repeated for an indefiniteperiod of time so long as the cooling system 202 is in operation.

[0056] It should be appreciated that the Tc's of some of the racks mayfall below the Tmin,set, whereas the Tc's of other racks may exceed theTmax,set. Thus, it should be appreciated that steps 310 and 312 may berespectively and substantially simultaneously performed on the variousracks.

[0057]FIG. 3B shows a flow diagram 350 of a second manner in whichanother embodiment of the invention may be practiced. The followingdescription of the flow diagram 350 is made with reference to the blockdiagram 260 illustrated in FIG. 2B, and thus makes reference to theelements cited therein. It is to be understood that the stepsillustrated in the flow diagram 350 may be contained as a utility,program, subprogram, in any desired computer accessible medium. Inaddition, the flow diagram 350 may be embodied by a computer program,which can exist in a variety of forms both active and inactive. Forexample, they can exist as software program(s) comprised of programinstructions in source code, object code, executable code or otherformats. Any of the above can be embodied on a computer readable medium,which include storage devices and signals, in compressed or uncompressedform.

[0058] Examples of computer readable storage devices includeconventional computer system RAM (random access memory), ROM (read onlymemory), EPROM (erasable, programmable ROM), EEPROM (electricallyerasable, programmable ROM), and magnetic or optical disks or tapes.Examples of computer readable signals, whether modulated using a carrieror not, are signals that a computer system hosting or running thecomputer program can be configured to access, including signalsdownloaded through the Internet or other networks. Concrete examples ofthe foregoing include distribution of the programs on a CD ROM or viaInternet download. In a sense, the Internet itself, as an abstractentity, is a computer readable medium. The same is true of computernetworks in general. It is therefore to be understood that thosefunctions enumerated below may be performed by any electronic devicecapable of executing the above-described functions.

[0059] In the flow diagram 350, steps 352-362 respectively correspond tosteps 302-312 recited hereinabove with respect to the flow diagram 300illustrated in FIG. 2A. Therefore, a detailed description of steps352-362 will not be made herein. Instead, one of ordinary skill in theart will readily recognize that the description made hereinabove withrespect to steps 302-312 has general applicability to steps 352-362 andmay thus be used interchangeably.

[0060] Therefore, beginning at step 364, the pressure of the coolingfluid returning to the cooling system 12 may be measured by a pressuresensor (e.g., the pressure sensor 56, 58, etc.). The measured pressuremay be relayed to the cooling system controller 246. The cooling systemcontroller 246 may determine whether the measured pressure is within apredetermined pressure range, e.g., a predetermined minimum set pointpressure (Pmin,set) and a predetermined maximum set point pressure(Pmax,set), at step 366. The predetermined pressure range may be setaccording to a maximum desired volume flow rate and/or velocity of thecooling fluid to be drawn in through the returns 38 a-38 c. In addition,the predetermined pressure range may be the substantial optimumoperating pressure desired for controlling the flow of cooling fluidthrough the returns. If the measured pressure is within thepredetermined pressure range, the cooling system controller 246 returnsto step 364. Furthermore, it is within the scope of the invention thatthe cooling system controller 246 may alternate between temperaturedependent control (e.g., steps 354 to 362) and pressure dependentcontrol (e.g., steps 364 to 372). In this regard, following step 366,the cooling system controller 246 may return to step 354 and followingthe step 356, the cooling system controller 246 may return to step 364.

[0061] If the measured pressure is not within the predetermined pressurerange, it is determined whether an absolute value of the measuredpressure (P) is below or equal to a minimum pressure set point(Pmin,set) at step 368. The absolute value is utilized because thepressure within the return is likely to be a negative value as comparedto the room pressure. In general, the predetermined pressure rangepertains to the threshold pressures to determine whether to increase ordecrease the movement of cooling fluid, e.g., in the space 16 and/orspace 22. The predetermined pressure range may be based upon a pluralityof factors, for example, a threshold operating pressure or range ofpressures that may be determined through testing to substantiallyoptimize the performance of the cooling fluid intake through the returns38 a-38 c.

[0062] If the absolute value of the P is determined to be below or equalto the Pmin,set, the cooling system controller 246 may operate toincrease the cooling fluid intake, e.g., by increasing the speed of thefan 24 at step 370. Otherwise, if the absolute value of the P isdetermined to exceed the Pmin,set, and thereby exceed the Pmax,set, thecooling system controller 246 may operate to decrease the intake of thecooling fluid, e.g., by decreasing the compressor capacity and/or thefan speed, at step 372.

[0063] Following steps 370 or 372, the cooling system controller 246returns to step 364. In addition, the steps following step 364 may berepeated for an indefinite period of time so long as the cooling system202 is in operation.

[0064] In accordance with an embodiment of the invention, the coolingrequirements within a data center may be analyzed to substantiallyoptimize the layout of the racks within the data center. In one respect,the substantial optimization of the rack layout in the data center mayenable the cooling system of the data center to operate at generallylower energy and greater efficiency levels by virtue of the reducedworkload placed on the components of the cooling systems, e.g.,compressors, fans, etc. The cooling requirements within the data centermay be analyzed by operation of any reasonably suitable commerciallyavailable computational fluid dynamics (CFD) tool, e.g., FLOVENT, a 3-Dmodeling software capable of predicting temperature variations basedupon fluid flows. By virtue of the numerical modeling, various airconditioning units as well as the vents described hereinabove may bepositioned throughout the data center to substantially control themanner in which the racks receive the cooling fluid. In addition, theair conditioning units may also be positioned to substantially maximizeand optimize their performances, e.g., to prevent one or more of the airconditioning units from being overworked.

[0065] In determining the cooling fluid distribution requirement withinthe data center, each of the racks may be assigned a heat load which maycorrespond to a maximum heat load predicted for that rack, e.g., throughanticipated power draw. For example, a rack containing 40 subsystems,e.g., computers, may have a maximum heat load of 10 KW and a rackcontaining 20 subsystems may have a maximum heat load of 5 KW. Byimplementing the CFD in this manner, for example in a data centercontaining 100 racks and four air conditioning units, racks having apotential for relatively larger heat loads may be relatively separatelylocated throughout the data center. In one respect, therefore, the airconditioning units within the data center may be operated atsubstantially less than maximum power levels and the racks may receivesufficient amounts of cooling fluid. More specifically, the powerrequired to operate the air conditioning units may be regulated toefficiently cool the fluid supplied to the racks by providingsubstantially only that amount of cooling fluid necessary to maintainthe racks within normal operating temperatures.

[0066] According to another embodiment of the invention, a CFD tool maybe implemented substantially continuously with the embodiments describedhereinabove with respect to FIGS. 1-3. More specifically, the CFD toolmay be utilized to substantially continuously vary the operation of thecooling system to operate according to the heat loads generated in theracks. In this regard, the anticipated or actual heat loads (e.g., basedupon the power draw of the components) on the racks may be inputted intothe CFD tool, along with one or more of the following properties:velocity of the cooling fluid flowing through various sections of thedata center and the distribution of temperature and pressure of thecooling fluid in the data center, to determine an optimal manner inwhich the air conditioning units may be operated as well as the flow ofthe cooling fluid through the vents to adequately cool the racks basedupon an analysis of the data center layout and the heat loads. The CFDtool may be implemented to produce a numerical model of the data centerto thus determine an optimized cooling distribution within the datacenter. A correlation of one or more of the following properties:velocity of the cooling fluid flowing through various sections of thedata center, distribution of temperature and pressure of the coolingfluid in the data center, and the power draw into the racks, may becreated based on the numerical modeling. The correlation may be used toinfer thermal conditions throughout the data center when only a minimumnumber of sensors are available during operation of the cooling system.In addition, the correlation may substantially reduce the amount of timerequired for the CFD tool to perform the computing operations.

[0067] Thus, for example, with respect to FIG. 3A, at step 312, anumerical model may be created to analyze an optimal manner in which thevolume flow and/or the velocity of the cooling fluid may be increasedwhile considering the effects of fluid flow from other racks. In thisrespect, based upon the analysis, the return configured to removecooling fluid from the vicinity of that rack and/or another return maybe caused to vary the volume flow and/or velocity of the cooling fluid.In addition, at step 314, the numerical model may be created todetermine whether the cooling system intake should be decreased basedupon the heat loads and the fluid flow throughout the data center. Forexample, if it is determined that a rack with an increasing heat loadmay receive a sufficient amount of cooling fluid by removing coolingfluid from a return generally away therefrom, the cooling system intakemay not be increased. Thus, by implementation of the CFD tool togenerally analyze the fluid flow characteristics and the temperatures ofthe racks, the amount of energy required to sufficiently cool the racksin the data center may be substantially optimized.

[0068] According to yet another embodiment of the invention, anyreasonable control system may be employed to control the cooling system202. Specific examples of control systems employable to control thecooling system 202 include, but are not limited to: agent based controland market based control. For example, in an agent based control system,each component of the cooling system 202 may be controlled by a software“agent” configured to interact and negotiate with the other agents toaccomplish a collective goal of cooling the racks in a most efficientmanner. In an example of a market based control system, each resource(e.g., cooling needs, fluid velocity, energy usage, etc.) may beassigned a unit value based on its relative supply and demand. The unitcosts of these resources may be negotiated via a known market basedsoftware application and units of these resources may be traded withinthe system.

[0069] By virtue of certain aspects of the invention, one of ordinaryskill in the art will readily recognize that the amount of energy, andthus the costs associated with cooling the racks located within a datacenter may be substantially reduced. In one respect, by operating thecooling system to circulate cooling fluid substantially only as neededby the racks, the cooling system may be operated at a relatively moreefficient manner as compared to conventional cooling systems.

[0070] What has been described and illustrated herein is an embodimentof the invention along with some of its variations. The terms,descriptions and figures used herein are set forth by way ofillustration only and are not meant as limitations. Those skilled in theart will recognize that many variations are possible within the spiritand scope of the invention, which is intended to be defined by thefollowing claims—and their equivalents—in which all terms are meant intheir broadest reasonable sense unless otherwise indicated.

1. A cooling system for cooling racks in a data center, said systemcomprising: a cooling device for circulating cooling fluid in said datacenter, said cooling device including a fan; a plenum having a pluralityof returns and an outlet, wherein said outlet of said plenum is in fluidcommunication with said fan, wherein said plurality of returns areconfigured for removing said cooling fluid from said data center and areoperable to vary a characteristic of said removal of cooling fluidthrough said returns, wherein said plurality of returns are configuredfor removing cooling fluid from a plurality of racks, said plurality ofracks respectively comprising enclosures configured to house a pluralityof PC boards.
 2. The system according to claim 1, wherein saidcharacteristic of said cooling fluid comprises at least one of volumeflow rate and velocity.
 3. The system according to claim 1, furthercomprising: at least one return controller operable to control at leastone of said returns, wherein said at least one return controller isconfigured to substantially independently control said returns tothereby substantially independently vary said characteristic of saidcooling fluid removal.
 4. The system according to claim 3, furthercomprising: a plurality of sensors configured to sense an environmentalcondition within said data center, said environmental conditionincluding at least one of temperature, humidity, pressure, and coolingfluid flow rate, wherein said at least one return controller isconfigured to substantially independently control said returns inresponse to said measured environmental condition. 5-50. (Canceled). 51.The system according to claim 1, wherein said characteristic of saidremoval of cooling fluid through said returns comprises the direction ofcooling fluid removal.
 52. The system according to claim 4, wherein saidplurality of sensors are configured to sense an environmental conditionin locations outside of the racks and wherein the at least onecontroller is configured to substantially independently control saidreturns in response to said measured environmental condition out side ofthe racks.
 53. The system according to claim 1, wherein the plurality ofreturns includes fans configured to draw cooling fluid from the datacenter.
 54. The system according to claim 53, wherein the fans aremovable, wherein a direction of cooling fluid removal is varied bymoving the fans.
 55. (Canceled).
 56. The method according to claim 69,wherein the various locations of said data center comprises a pluralityof racks.
 57. The method according to claim 69, wherein the step ofvarying said removal of said cooling fluid from said racks in responseto said sensed temperatures being below or equal to the predeterminedminimum set point temperature comprises varying the direction of removalof said cooling fluid.
 58. The apparatus according to claim 73, whereinthe various locations of said data center comprises a plurality ofracks.
 59. The apparatus according to claim 73, wherein the means forvarying said removal of said cooling fluid from said racks in responseto said sensed temperatures being below or equal to said predeterminedminimum set point temperature comprises means for varying the directionof said removal of said cooling fluid.
 60. The computer readable mediumaccording to claim 40, wherein the various locations of said data centercomprises a plurality of racks.
 61. The computer readable mediumaccording to claim 77, wherein the step of varying said removal of saidcooling fluid from said racks in response to said sensed temperaturesbeing below or equal to the predetermined minimum set point temperaturecomprises varying the direction of removal of said cooling fluid. 62-67.(Cancelled).
 68. The system according to claim 1, wherein the pluralityof returns are independent of the racks.
 69. A method of cooling aplurality of racks in a data center, said method comprising: activatinga cooling system and opening a plurality of returns in fluidcommunication with a plenum, said plenum being in fluid communicationwith a cooling system, said returns being configured to remove coolingfluid from various locations of said data center; sensing thetemperatures of said racks, wherein the racks comprise discreteenclosures separate from one another; determining whether said sensedtemperatures are within a predetermined temperature range; and varyingsaid removal of said cooling fluid from said racks in response to saidsensed temperatures being outside said predetermined temperature range.70. The method according to claim 69, further comprising: determiningwhether the measured temperatures of said racks are below or equal to apredetermined minimum set point temperature; decreasing the removal ofsaid cooling fluid from locations around said racks having measuredtemperatures that fall below or equal said predetermined minimum setpoint temperature; and increasing the removal of said cooling fluid fromsaid racks having measured temperatures that exceed said predeterminedminimum set point temperature.
 71. The method according to claim 70,further comprising: decreasing an intake of said cooling fluid by saidcooling system in response to said decrease in cooling fluid removalfrom said racks exceeding said increase in cooling fluid removal fromsaid racks.
 72. The method according to claim 70, further comprising:increasing an intake of said cooling fluid by said cooling system inresponse to said decrease in cooling fluid removal from said racksfalling below said increase in cooling fluid removal from said racks.73. An apparatus for cooling a plurality of racks in a data center, saidapparatus comprising: means for activating a cooling system and openinga plurality of returns in fluid communication with a plenum, said plenumbeing in fluid communication with a cooling system, wherein each of saidreturns is configured to remove cooling fluid from various locations ofsaid data center; means for sensing the temperatures of said racks,wherein the racks comprise discrete enclosures separate from oneanother; means for determining whether said sensed temperatures arewithin a predetermined temperature range; and means for varying saidremoval of said cooling fluid from said racks in response to said sensedtemperatures being outside said predetermined temperature range.
 74. Theapparatus according to claim 73, further comprising: means fordetermining whether the measured temperatures of said racks are eachbelow or equal to a predetermined minimum set point temperature; meansfor decreasing the removal of said cooling fluid from locations aroundsaid racks having measured temperatures that fall below or equal saidpredetermined minimum set point temperature; and means for increasingthe removal of said cooling fluid from said racks having measuredtemperatures that exceed said predetermined minimum set pointtemperature.
 75. The apparatus according to claim 74, furthercomprising: means for decreasing an intake of said cooling fluid by saidcooling system in response to said decrease in cooling fluid removalfrom said racks exceeding said increase in cooling fluid removal fromsaid racks.
 76. The apparatus according to claim 74, further comprising:means for increasing an intake of said cooling fluid by said coolingsystem in response to said decrease in cooling fluid removal from saidracks falling below said increase in cooling fluid removal from saidracks.
 77. A computer readable medium on which is embedded computersoftware, said software comprising executable code for performing amethod of cooling a plurality of racks in a data center, said methodcomprising: activating a cooling system and opening a plurality ofreturns in fluid communication with a plenum, said plenum being in fluidcommunication with a cooling system, each of said returns beingconfigured to remove cooling fluid from various locations of said datacenter; sensing the temperatures of said racks, wherein the rackscomprise discrete enclosures separate from one another; determiningwhether said sensed temperatures are within a predetermined temperaturerange; and varying said removal of said cooling fluid from said racks inresponse to said sensed temperatures being outside said predeterminedtemperature range.
 78. The computer readable medium according to claim77, further comprising: determining whether the measured temperatures ofsaid racks are below or equal to a predetermined minimum set pointtemperature; decreasing the removal of said cooling fluid from locationsaround said racks having measured temperatures that fall below or equalsaid predetermined minimum set point temperature; and increasing theremoval of said cooling fluid from said racks having measuredtemperatures that exceed said predetermined minimum set pointtemperature.
 79. The computer readable medium according to claim 78,further comprising: decreasing an intake of said cooling fluid by saidcooling system in response to said decrease in cooling fluid removalfrom said racks exceeding said increase in cooling fluid removal fromsaid racks.
 80. The computer readable medium according to claim 78,further comprising: increasing an intake of said cooling fluid by saidcooling system in response to said decrease in cooling fluid removalfrom said racks falling below said increase in cooling fluid removalfrom said racks.